GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job by raulcd · Pull Request #50406 · apache/arrow

raulcd · 2026-07-07T06:53:46Z

Rationale for this change

Running a PyArrow Windows build with Assertions enhances our coverage, keeping debug information can helps us on future investigations.

What changes are included in this PR?

Run Windows CI job PyArrow build with RelWithDebInfo but remove NDEBUG so assertions are exercised.
We can't use DEBUG directly because otherwise we require linking with CPython debug build (Py_Debug).

Are these changes tested?

Yes on CI, I've validated the flags for building are correct.

Are there any user-facing changes?

No, this is only for our CI job.

GitHub Issue: [CI][Python] Use Debug build on our Windows Python CI job #47583

… job

github-actions · 2026-07-07T07:00:19Z

⚠️ GitHub issue #47583 has been automatically assigned in GitHub to PR creator.

raulcd · 2026-07-07T11:49:37Z

@pitrou the problem seems to be that Debug builds try to link python313_d.lib instead of python313.lib. This seems to be defined upstream on Python itself here (link for 3.16 but 3.13 is the same):
https://github.com/python/cpython/blob/2cd5b79284dd2331cbd9e11afabbfbf7e906103d/PC/pyconfig.h#L340-L342
And it fails with:

 LINK : fatal error LNK1104: cannot open file 'python313_d.lib'

On the Windows CI job we use actions/setup-python which do not allow to install CPython debug. I could try RelWithDebInfo instead of DEBUGotherwise I think we require installing CPython on Debug mode (manually?) which would not allow us to use the setup-python action.
Do you have other ideas?

pitrou · 2026-07-07T13:17:44Z

Ouch, I had missed that _DEBUG implied Py_DEBUG. This is annoying: we would like our own assertions (and the C++ stdlib's) without needing a special build of CPython :(

pitrou · 2026-07-07T13:21:03Z

See https://discuss.python.org/t/building-debug-version-of-extensions-on-windows-without-debug-binaries/105012

pitrou · 2026-07-07T13:30:13Z

According to the discussion above, perhaps we should use RelWithDebInfo (or even plain Release) and undefine the C preprocessor macro NDEBUG.

pitrou · 2026-07-07T13:36:13Z

I think we require installing CPython on Debug mode (manually?) which would not allow us to use the setup-python action

We can of course do this, assuming we find a readily available debug build of CPython on Windows, but then some dependencies might need to be recompiled from source and we'd rather avoid that.

raulcd · 2026-07-08T15:07:54Z

This seems to be good, on this PR:

[26/94] "C:\Program Files\Git\usr\bin\ccache.exe" C:\PROGRA~~1\MICROS~~2\2022\ENTERP~~1\VC\Tools\MSVC\1444~~1.352\bin\Hostx64\x64\cl.exe /nologo /TP -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_PYTHON_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D__SSE2__ -D__SSE4_1__ -D__SSE4_2__ -Darrow_python_EXPORTS -ID:\a\arrow\arrow\python\pyarrow\src -IC:\Users\runneradmin\AppData\Local\Temp\tmpxly05kia\build\pyarrow\src -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Include -I"C:\Program Files\Git\usr\include" -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Lib\site-packages\numpy_core\include /EHsc /wd5105 /bigobj /utf-8 /W3 /wd4800 /wd4996 /wd4065 /arch:SSE4.2 /DWIN32 /D_WINDOWS /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /wd4190 /wd4293 /wd4800 /wd4551 /O2 /Ob1 -std:c++20 -MD -Zi /showIncludes /FoCMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj /FdCMakeFiles\arrow_python.dir\ /FS -c D:\a\arrow\arrow\python\pyarrow\src\arrow\python\arrow_to_pandas.cc

No /DNDEBUG and debug symbols present -Zi.

Previously for Release /DNDEBUG was present and -Zi was not:

[25/94] "C:\Program Files\Git\usr\bin\ccache.exe" C:\PROGRA~~1\MICROS~~2\2022\ENTERP~~1\VC\Tools\MSVC\1444~~1.352\bin\Hostx64\x64\cl.exe /nologo /TP -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_PYTHON_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D__SSE2__ -D__SSE4_1__ -D__SSE4_2__ -Darrow_python_EXPORTS -ID:\a\arrow\arrow\python\pyarrow\src -IC:\Users\runneradmin\AppData\Local\Temp\tmpgjv_4rcf\build\pyarrow\src -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Include -I"C:\Program Files\Git\usr\include" -IC:\hostedtoolcache\windows\Python\3.13.14\x64\Lib\site-packages\numpy_core\include /EHsc /wd5105 /bigobj /utf-8 /W3 /wd4800 /wd4996 /wd4065 /arch:SSE4.2 /DWIN32 /D_WINDOWS /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /wd4190 /wd4293 /wd4800 /wd4551 /O2 /Ob2 /DNDEBUG -std:c++20 -MD /showIncludes /FoCMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj /FdCMakeFiles\arrow_python.dir\ /FS -c D:\a\arrow\arrow\python\pyarrow\src\arrow\python\arrow_to_pandas.cc

Copilot

Pull request overview

This PR updates the Windows PyArrow CI build configuration to use RelWithDebInfo while ensuring assertions remain enabled (avoiding a full Debug build that would require linking against CPython debug libraries).

Changes:

Switch Windows C++ build type from Release to RelWithDebInfo.
Override MSVC RelWithDebInfo compile flags to avoid defining NDEBUG (so assertions are exercised).
Pass the same CMake build type and flag overrides through the pip install build configuration for PyArrow.

pitrou · 2026-07-09T08:06:42Z

Can you set CMAKE_MSVC_DEBUG_INFORMATION_FORMAT to Embedded to make ccache effective here?

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

raulcd · 2026-07-09T10:34:55Z

Can you set CMAKE_MSVC_DEBUG_INFORMATION_FORMAT to Embedded to make ccache effective here?

Done, this should be ready for review @pitrou

pitrou · 2026-07-09T11:27:31Z

Thanks! Total runtime for the Windows CI job went down from 43m to 16m thanks to caching!

apacheGH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI…

8f4d7ec

… job

github-actions Bot added the awaiting committer review Awaiting committer review label Jul 7, 2026

raulcd added 2 commits July 8, 2026 16:03

Use RelWithDebInfo and try to strip -DNDEBUG

f86e22e

Fix comment to be more accurate

136c38d

raulcd changed the title ~~GH-47583: [CI][Python] Enable PyArrow Debug build on Windows CI job~~ GH-47583: [CI][Python] Enable PyArrow RelWithDebInfo build with assertions on Windows CI job Jul 8, 2026

raulcd marked this pull request as ready for review July 8, 2026 15:08

raulcd requested a review from assignUser as a code owner July 8, 2026 15:08

Copilot AI review requested due to automatic review settings July 8, 2026 15:08

raulcd requested review from jonkeane and kou as code owners July 8, 2026 15:08

Copilot started reviewing on behalf of raulcd July 8, 2026 15:08 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread ci/scripts/python_build.bat

Comment thread ci/scripts/python_build.bat

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 8, 2026

raulcd requested a review from pitrou July 8, 2026 15:41

Add CMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded as requested

d15989c

Copilot AI review requested due to automatic review settings July 9, 2026 08:30

Copilot started reviewing on behalf of raulcd July 9, 2026 08:31 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 9, 2026

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread ci/scripts/python_build.bat

github-actions Bot removed the awaiting change review Awaiting change review label Jul 9, 2026

github-actions Bot added the awaiting changes Awaiting changes label Jul 9, 2026

pitrou approved these changes Jul 9, 2026

View reviewed changes

raulcd merged commit 98a92c2 into apache:main Jul 9, 2026
48 checks passed

raulcd removed the awaiting changes Awaiting changes label Jul 9, 2026

raulcd mentioned this pull request Jul 9, 2026

[CI][Python] Use Debug build on our Windows Python CI job #47583

Closed

raulcd deleted the GH-47583 branch July 9, 2026 12:48

Uh oh!

Conversation

raulcd commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

raulcd commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Jul 7, 2026

Uh oh!

pitrou commented Jul 7, 2026

Uh oh!

pitrou commented Jul 7, 2026

Uh oh!

pitrou commented Jul 7, 2026

Uh oh!

raulcd commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

raulcd commented Jul 9, 2026

Uh oh!

pitrou commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raulcd commented Jul 7, 2026 •

edited

Loading

raulcd commented Jul 7, 2026 •

edited

Loading